Speech-Enabled Augmented Reality Supporting Mobile Industrial Maintenance

A maintenance technician who talks to pumps and pipes might not seem like the ideal person to entrust with keeping a factory running smoothly. However , realizing such an industrial sce-nario—that is, opening communication through augmented reality—could empower the industrial-maintenance industry with immeasurable freedom and efficiency. To do so, we must utilize numerous technologies. First, we could locate a set of uniquely recognizable visual markers at different locations around a factory. As maintenance engineers move with either a wearable or lightweight notebook computer equipped with an integrated camera , a speech-enabled AR framework could employ vision-based algorithms to identify the markers and localize their positions in real-world 3D space. When the technicians enter a specific component's vicinity, the framework could then automatically engage them in a context-specific speech dialog concerning the corresponding factory component, as Figure 1 shows. Finally, we could integrate the framework with plant automation software to facilitate convenient real-time information access. For example, technicians approaching a container tank could ask, " Current status? " The container tank might reply, " 34 percent full of water at a temperature of 62 degrees Celsius. " Service and maintenance are by necessity mobile activities. To address this environment's needs, Siemens Corporate Research (SCR) has been applying 3D interaction and visualization techniques to industrial automation. Our research focuses specifically on improving automated support for mobile environments. Navigation and interaction in 3D environments can become complex for the user. So, we have developed a speech-enabled AR (SEAR) framework for experimenting with the seamless navigation of convenient, context-sensitive, rich multi-modal user interfaces for mobile maintenance. Our approach uses all the technologies we mention in the proposed scenario. To the best of our knowledge, ours is the first reported framework running on a mobile device that offers a multimodal user interface that synchronizes a 3D AR graphical view with a location-sensitive 3D speech-driven interface. (See the " Related Work " sidebar for more information on other research and activities.) Figure 2 shows a high-level functional view of the SEAR system. The inputs are on the left, the outputs are on the right, and wireless communication with the plant management system is below. SEAR comprises a 3D Virtual Reality Markup Language component and speech-recognition and speech-synthesis engines to enable a spoken dialogue with complex 3D objects in a scene-and vision-based marker-tracking system. This system facilitates seamless navigation. The 3D objects representing the industrial equip-The SEAR …